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ABSTRACT 

With limits on both personnel and time available to conduct 
effective instruction, the decision is being made increasingly to enhance 
instructor-led courses with Computer-Based Training (CBT) . The effectiveness 
of this conversion is often unknown and in many cases empirical evaluations 
are never conducted. This paper describes and discusses the evaluation and 
effectiveness of adaptive courseware authoring and utilization in the context 
of the Submarine Officer Basic Course (SOBC) at the Naval Submarine School, 
Groton, CT . The Naval Submarine School (NAVSUBSCOL) , the "Center of 
Excellence" for Submarine Warfare Training, has recently dedicated a new, 
state-of-the-art electronic training facility that hosts ten Advanced 
Electronic Classrooms (AEC) , a building-wide enterprise Classified LAN 
(CLAN), and external high bandwidth connectivity. The benefits of 
implementing adaptive CBT in the Submarine School range from possible time 
savings in specific curricula to automated remediation and instruction. 
Manpower savings- will permit Submarine School to pursue implementation of a 
Distance Learning support division comprised of experienced subject matter 
experts who have instructional expertise in front of a class and as CBT 
authors. CBT cannot replace the instructor completely. Students asserted 
uniformly that the face-to-face communications were at times critical for the 
understanding of particularly complex concepts. Training via CBT provides 
exceptional flexibility to both the schoolhouse (in terms of resource 
scheduling) and to the student (to learn at his convenience) . (Author) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



IR021704 



An Empirical Evaluation of Sonar Courseware Developed with 
Intelligent Tutoring Software (InTrain™) at Naval Submarine School 



A' 



VO 

o^ 

VO 

r- 



Q 



W 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



G.H. Marks 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Marcy Birchard 
Sonalysts, Inc. 

2 1 5 Parkway North 
Waterford, CT 06385 
birchard@sonalvst$.com 

Charles Dye 
Sonalysts, Inc. 

2 1 5 Parkway North 
Waterford, CT 06385 
csdve@sonalvsts.com 

LCDR John Gordon 
Naval Submarine School 
Groton, CT 06340 

LCDR‘-John.Gordon@cnet.navv.mil 



OF EDUCATION 

pni A ^®search and Improvement 

educational resources information 

U CENTER (ERIC) 

This document has been reproduced as 
' rece.ved from the person or organizalfon 
originating it. ^ 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Abstract: With limits on both personnel and time available to conduct 

effective instruction, the decision is being made increasingly to enhance instructor-led 
courses with Computer-Based Training (CBT), The effectiveness of this conversion is 
often unknown and in many cases empirical evaluations are never conducted. This paper 
describes and discusses the evaluation and effectiveness of adaptive courseware 
authoring and utilization in the context of the Submarine Officer Basic Course (SOBC) at 
the Naval Submarine School, Groton, CT, 



Introduction 

The Naval Submarine School (NAVSUBSCOL), the “Center of Excellence” for Submarine 
Warfare Training, has recently dedicated a new, state-of-the-art electronic training facility that hosts ten 
Advanced Electronic Classrooms (AEC), a building-wide enterprise Classified LAN (CLAN), and external 
high bandwidth connectivity. The NAVSUBSCOL Officer Training Department provides the Fleet with 
Prospective Commanding Officers (PCOs), Prospective Executive Officers (PXOs), Department Heads, 
Division Officers, and a host of officer billet and specialty training courses. NAVSUBSCOL training 
responsibilities range from engineering, damage control, and ship control, to combat training simulation 
and tactical employment of the submarine. Currently, the curricula are based primarily on instructor-led 
presentations and practical exercises in lab/simulator environments. 

Recently, NAVSUBSCOL determined that the demand for training was outpacing the available 
resources (instructor-led presentations and practical exercises). To address the situation, NAVSUBSCOL 
searched for a supplement to the instructor-led training by adding computer-based training (CBT). The goal 
was to leverage the available resources of the lead instructors while incorporating additional benefits of 
current courseware, maintaining internal control of content, and reduce staff loading. 

This case study discusses the efforts made at NAVSUBSCOL to integrate CBT into the 
curriculum and the effects this type of learning will have on learning outcomes. Through a comparison 
study of both CBT and instructor-led courseware, supportive evidence can be garnered that support a 
savings in development time and improvements in test scores. 

Traditional multimedia instruction creates an engaging learning environment in which multiple 
sensory pathways convey information to the student. NAVSUBSCOL wanted to choose an authoring tool 
that would allow instructors to re-use content and learning objectives throughout multiple courses, while 
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delivering individualized instruction to a large volume of students. Likewise, it wanted the authoring tool to 
be easy to use and require low developmental times due to constant turnover of personnel. 

InTrain™ provides such a tool. InTrain instruction builds and tailors the content and form of 
instruction to the moment-to-moment needs of the student. This results in faster acquisition and longer 
retention than is seen with conventional learning approaches. 

In a marked contrast to conventional CBT, InTrain develops an individual education plan (lEP) 
for each student and makes its instructional decisions at run-time. Students proceed when they demonstrate 
that they are meeting the goals of instruction mastery. Students who need more time or different methods to 
reach the same level of performance receive the instruction they need. 

While forming an lEP, InTrain considers information about the student (e.g., the student's mastery 
profile and instructional history) and about the goals of the curriculum (/.e., which objectives must the 
student meet at what level of proficiency). The lEP describes the order in which the system will introduce 
new topics and the training techniques that will support the introduction. When assessment is needed (both 
pre- and post-tests may be used), InTrain randomly selects among the assessment options (e.g., test 
questions) associated with the learning objectives under investigation. Similarly, when InTrain wants to 
present instruction, it selects among the instructional options associated with the target learning objectives. 
Instruction and assessment continue until the student has adequately mastered all of the required learning 
objectives. 

Shorter development times are also addressed by the InTrain Authorl software {InTrain Authorl is 
the interface through which InTrain courseware is authored. InTrain does not require any programming 
experience and offers an “as you go” tutorial on content development. This ease of use allows for shorter 
development time and less “training” time for new courseware development. 



Method 

To assess the effectiveness of the training to be demonstrated, Sonalysts, Inc. conducted a study in 
conjunction with NAVSUBSCOL. Traditional instructor-led classroom students were given 8.5 hours of 
instruction about SONAR, as were the CBT students. The learning objectives and goals of the content 
presented to each group was the same. In contrast, another group of students received InTrain instruction. 
Both groups were given the same amount of time to interact with the content. 

After the second group completed the curriculum, both groups were then given a final written 
examination, comprising of 80 questions. Instructors scored the tests that included multiple-choice and 
true/false questions. The students needed to achieve a minimum of 70% to pass the course. The scores of 
the instructor-led group, were in-line with historical, satisfactory averages for this content and type of 
instruction. 

Traditionally, training for the NAVSUBSCOL has taken place in instructor-led classrooms. Each 
classroom comprises approximately 24 students, who spend a portion of a 10-week course learning passive 
sonar theory and underwater acoustics. The focus of the comparison is on a specific 8.5 hours of the Sonar 
curriculum taught during the ten-week course. The 8.5 hours of instruction cover the following courses: 
Passive Sonar Equation, Total Background Noise, Beamforming, Figure of Merit, Source Level, and 
Propagation Loss. 

While each course has a standardized set of Learning Objectives and a Curriculum Outline, each is 
modified accordingly. Courseware for traditional instruction has usually been developed in PowerPoint 
lessons with Instructor Discussion Points and Lessons Learned being conveyed by each instructor. 
PowerPoint are inherited by the previous lesson instructor and amended when the need occurs. 

Students are scheduled for each of the above topics based on numbers of students, classroom 
availability, and instructor availability. Delivery time for each course is a pre-defmed amount and 
scheduled by a central person. Students are assigned to the next course as they move through the 
curriculum. At the end of each component of instruction (sonar, tactics, plots, etc.) students are given a test 
comprising 80 questions. 

In contrast to traditional methods, CBT was developed. To address NAVSUBSCOL’s goal of 
authoring CBT “in house,” Submarine School staff instructors, working with guidance from instructional 
designers and subject matter experts from Sonalysts, utilized InTrain AUTHORl to develop an eight-hour, 
adaptive CBT course on the passive sonar theory equation, theories, and underwater acoustics. This 
development effort was a multi-stage process. Initially instructors were given a basic INTRO courses in 
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courseware development and creating computer-based training. Instruction focused on creating and 
decomposing learning objectives, creating assessment items, and developing content (at a variety of 
targeted levels). Ultimately, one instructor became responsible for development of 8.5 hours of SONAR 
courses. InTrain allows for instructional method neutrality. The tool does not require one to build/create 
Sharable Content Objects/presentations in a particular manner, thereby allowing the developer complete 
freedom in content presentation and delivery. Sonalysts served as “consultants” to the instructor in charge 
of development. Media was reused from courseware previously authored for other topics, instructor-led 
training, and Submarine Officer On-Board Training (SORT) courses. 

The content developed using InTrain, was delivered to a class of 21 students that utilized the 
courseware during two four-hour dedicated periods, plus a one-hour wrap-up following the testing. The 
courseware was parsed into six topics that mimicked the training methodology of the material in instructor- 
led instruction. The CBT was deployed using options that permitted the student to learn the material and 
demonstrate mastery through assessment. Students were permitted two attempts at the assessments. If the 
student failed a second assessment, an instructor-led remediation was conducted. 

Students were given the same amount of time to learn using the computer-based course as they 
were given to learn through the traditional methods. The reasons for this were two-fold: 1) Delivery of 
CBT was an emerging concept within NAVSUBSCOL. While computer-based training is available, the 
courses are self-paced auxiliary learning on Lessons Learned and Practice for defined skills. There was no 
model from which to build time/scheduling differences around this type of learning. 2) The SOAC/SOBC 
Associate Assistant Director of Officer Training maintained that it was important that an appropriate 
amount of time was scheduled for students to receive the training. The most efficient manner to achieve 
this was to “schedule” it in the same blocks of time as the traditional methods of delivery. 

Upon completion of the material, students were able to review/return to the material at their 
convenience for follow-on study. 

There were four measures by which results were garnered for this comparison study: Survey, Test, 
Historical Averages, and Development Time. 

Survey - At the conclusion of the computer-based instruction {InTrain Author!), students and 
instructors were given a Level 1 evaluation. This survey consisted of both open-ended questions and 
Likert-scale questions. 

Test - The paper-based test included 80 questions that were both multiple-choice and true/false. 
Both groups of students (CBT and instructor-led) were given the same test and graded using the same set of 
instructors. 

Historical Averages - The historical averages used in the study are the averages of students who 
had gone through the instructor-led training in the past two years. The scores were from a slightly revised 
version of the test given to these students (The revision included the learning objectives but a few questions 
were changed). 

Development Time - Development time for the InTrain course was measured. 

Results 

The results of both courseware development and actual “testing” of the courseware as indicated 
below show improved in results in both the authoring content and the student performance. 



Survey 



The results of the students’ survey were interesting. A 5-point Likert scale was used for the 20 
questions. 

The areas that scored highest (4-4.5) were the presentation of the material and the way in which 
students interacted with the course. A score of 4.2 was given to the overall usefulness of the course within 
the curriculum in general. 

The students’ response to another set of questions (scoring 2. 6-2. 9) demonstrated that no 
additional instruction or assessment was necessary, lending credibility to the individualized component of 
the training which provides “just enough” instruction to achieve mastery of the learning objective. There 
were enough assessment and instruction to teach the topic without being overwhelmed by obscure 
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materials. This method targets the learner to specific information to support specific concepts and 
objectives. 



Test Results 

Students’ scores in both the instructor- led group and the InTmin group were measured and 
compared on the basis of results from the 80-question test. 





Instructor-Led Group 
(Last 2 classes of 
students) 


InTrain Students 
(One class) 


Historical^ 

(Instructor-Led 

Group) 


Population 


75 


21 


>300 


Student Mean^ 


88.5% 


95% 


84.5% 


No. of Exam Failures^ 


3 


0 


N/A 



Table 1. Evaluation Results 



Courseware Development 

No metrics exist to gauge the amount of development time required to prepare instructor-led 
courseware because the instructor-led materials are inherited from the original instructor and revised over 
several years. 

Although CBT development ratios can vary widely, actual development time was compared to a 
benchmark of 200 hours per hour of CBT. For this effort, that would yield an approximate effort of 1600 
hours of authoring time to design, storyboard, implement, and test the course. 

In contrast to this benchwork, the Submarine School staff spent a total of 500 hours for the 
development effort. Sonalysts’ instructional designers and subject matter experts spent an additional 150 
hours providing feedback and guidance on effective CBT design and content presentation. 



Discussion 

The survey results yielded some interesting information. The open-ended questions asked 
throughout the survey indicated that the students were favorably impressed with and receptive to the CBT 
that was presented. Additionally, they liked the “individualized instruction” that was presented through 
InTrain allowing some of the students to finish quicker and more efficiently than others who required more 
time or instructor intervention. Some negative responses about the CBT included the need to have an 
instructor review session at the conclusion of training (before the test). Another negative comment was the 
length of time dedicated to each training session (taught via CBT) was too long. In both cases each 
comment can be resolved in a simple and effective manner. 

NAVEDTRA 131 (Task Based Instruction), the controlling doctrine for the course in quesiton, 
requires that all instruction that results in a bottleneck be serially scheduled. That means, for a class of 12 
students participating in a learning experience that takes 4 hours and has capacity for only 6 students, an 
entire training day must be scheduled against it. By its nature, CBT provides unparalleled flexibility to 
obviate bottleneck scheduling. Students learn at their own pace, on their own schedule. In an instant 
example, the other six students not in the learning experience can complete assigned CBT required for 
another part of the course. The time saved by removing bottleneck scheduling is approximately 24 hours. 

If given proper integration into the NAVSUBSCOL curriculum scheduling process, students 
would proceed through the courses on an individualized basis. In fact, they would be assigned courses and 
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a completion date and proceed at their own pace toward that completion date. This is the way that 
NAVSUBSCOL is moving with the integration of a Learning Management System (LMS) to create such 
assignments. 

More CBT is undergoing development in other areas of the 10-week curriculum matrix. When 
more of the courses are converted to CBT and an entire curriculum can address scheduling, changes will be 
easier to implement. When timesaving is realized, an instructor-led review session can be addressed and 
implemented to quantify knowledge gained through the CBT. 

Additionally, the study yielded anecdotal evidence that four hour training periods for the students 
were too long and resulted in student “burn-out,” an assertion supported by student feedback following the 
training. A consensus of input suggests that a period of two hours at any given time/topic is the longest that 
should be scheduled. The artificiality introduced by the beta test (i.e., the test dictated two four-hour 
periods of study) required the student not study when they were ready to, but rather when the test required 
it. As such, student mastery and navigation through the material suffered at the end of the first training 
period due to student fatigue. 

Finally, we address test scores. We can see that the scores on the test were higher for the InTrain 
classes than the traditional instructor-led method. Since both groups were achieving at a high-level, the 
observed results might actually understate the size or effect one would expect in other settings. Another 
factor that influenced the “ceiling effect” introduced by these scores is the high average among the 
students. If the tests were longer and students had been scoring in the 70%, we would have expected to see 
an even greater range of effectiveness defined. 



CBT Benefits 

The CBT in question does provide some benefits when measuring interaction time metrics: In the 
study, the students required an average of 8.2 hours to complete the 6 topics in the CBT. In comparison, 
instructors require 8.5 hours to teach the same material in an instructor- led lecture format. One would 
assume that the CBT seat time would be reduced about 50-60%. However, given the restrictions of the 
imposed time schedule this was not seen. One would assume that the deployment time would be less once a 
new scheduling system is implemented and appropriate time values are added to the scheduling matrix to 
accommodate Computer-Based Courses. 

Additionally, there was a time-savings in terms of instructor remediation time. A total of 7 
students required remediation by an instructor, yielding a total of approximately 1.5 man-hours of 
instructor time for student instruction (vice the nominal 8.5 hours required to teach the material plus any 
time spent in individual tutoring). For the beta test. Submarine School instructors attended all training 
sessions, but it is envisioned that once CBT is well entrenched in the SOBC curriculum and students 
understand how to use it, direct instructor supervision will not be required. Rather, CBT will be employed 
to teach basic theory and concepts, and instructor led seminars synthesizing material and covering more 
advanced concepts will be used to periodically ensure students have satisfactorily advanced through the 
material. 

On its face, the integration of Sonalysts’ tool represents a 59.3% savings in development time over 
a typical CBT effort. However, some elements in the study influence this apparent savings: 

• The Topic Learning Objectives derived from course Personal Performance Profiles (PPP) tables 
already existed; hence the course was not built “from scratch.” 

• The Submarine School staff author had no training in CBT development. While this is a de 
rigueur requirement for any authoring tool fielded for Submarine School, it is also true that those 
same personnel will become more proficient in the use of the tool as more CBT is authored. 

• . A limited amount of media (3 graphics) was produced in support of this effort. For the remaining 

approximately 700 graphics and animations used, existing media and or local generation tools 
were leveraged for the media production. It is anticipated that during a typical CBT authoring 
effort, the existing media bank will not be quite as well populated. 

• The purpose of the study was also to test InTrain AUTHOR! Several software problems were 
identified and corrected during the test. This resulted in an aggregate reduction in authoring 
capability for a period of about one week. InTrain has completed Beta testing and made changes, 
so this type of delay should no longer occur. 
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Conclusions 



The entire authoring effort, deployment of the course, and post-CBT analysis spanned 3.5 months. 
Sonalysts, provided the IT, instructional design, and subject matter expertise in an effort to investigate the 
usefulness of InTrain CBT in an established, structured U.S. Navy curriculum. Submarine School provided 
exceptional dedication to the effort, including the combined work of the CBT author and testing team. 

The InTrain CBT test provided valuable input to the product developers at Sonalysts, both in 
terms of performance and study recommendations. As part of the implementation of distance learning 
efforts at NAVSUBSCOL, Sonalysts will be involved with the integration of the submarine officer training 
curricula into the THINQ'^*^ LMS (Training Server). This initial testing allowed the software engineering 
staff at Sonalysts to evaluate the requirements and use patterns of CBT (and specifically InTrain) in an 
enterprise LAN environment. ^ 

As part of the population of the LMS with content, Submarine School must develop and deploy a 
robust web-hosted CBT course structure to allow remote users access to training prerequisites and 
remediation to support both professional development and continuing training onboard submarines. This 
test provided the Submarine School staff some experience in CBT design and provided some metrics and 
recommendations about implementing CBT to maximize its benefits to the student population. 



Summary 

Based on student performance and feedback received, the following conclusions can be drawn 
from the test: 

• The benefits of implementing adaptive CBT in the Submarine School range from possible 
timesavings in specific curricula to automated remediation and instruction. Manpower savings will 
permit Submarine School to pursue implementation of a Distance Learning support division 
comprised of experienced subject matter experts who have instructional expertise in front of a 
class and as CBT authors. 

• CBT cannot replace the instructor completely. Students asserted uniformly that the face-to-face 
communications were at times critical for the understanding of particularly complex concepts. 
While not applicable to the material delivery in the Beta test, NAVSUBSCOL is planning on 
maintaining some instructor-led seminars to provide higher-level synthesis of read material. 

• Training via CBT provides exceptional flexibility to both the schoolhouse (in terms of resource 
scheduling) and to the student (to learn at his convenience). 

The potential to improve training efficiencies, effectiveness, and achieve Fleet readiness 
improvements with the infusion of advanced training methodology and technology is enormous and clearly 
measurable. This effort is not envisioned to replace the resident training experience but to enhance the level 
of professional development that can be achieved and to provide continuous training and maintenance of 
readiness to students and groups ashore or at sea. A substantial commitment to investing in information 
technology infrastructure has been made, but the return on investment will not be realized without a 
concomitant fielding of advanced learning methodology products. It is recognized that this vision in its 
many forms and variations cannot be achieved immediately; the first small steps will reflect changes in 
technology and processes. 



Historical averages are based on exam results from the last year. 

2 

The written examination included two topics that were not developed into CBT but were presented to the students in 
traditional lecture format. The average for those two questions was 79%. A corrected average for performance on just 
the InTrain instructed material was 96.2%. Removal of these two topics from the other tests did not change the 
historical or last 4 class means in any significant fashion (+ 0.2%). 

Exam failures are counted in the student mean as 70%. Remediation and re-exam are conducted upon exam failure. 
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